1. Field of the Invention
The present invention relates generally to a computerized automation system and method, and more specifically, to a computerized automation system and method which utilize a matrix or array of vectors specifying actions to be performed and expected results of those actions, whereby to permit comparison between the expected and actual results of these actions. Although particular utility for the present invention is found in the field of computer network test automation, and the present invention will be described herein in connection with such utility, as will be appreciated by those skilled in the art, other utilities are also contemplated for the present invention, including other automated programming applications.
2. Brief Description of Related Prior Art
Many automated systems/procedures currently exist for stimulus/response testing of various units-under-test (UUT), such as printed circuit boards, computer networks, and other electronic systems and devices. Examples of such systems include those devoted to boundary-scan, bed of nails, built-in self test, and other techniques. In general, such testing involves supplying test input stimuli (e.g., commands, data, etc.) to the UUT, observing actual outputs (e.g., actions, data, etc.) generated by the UUT in response to the stimuli, and comparing the actual outputs to outputs that are expected if the UUT is functioning properly. Depending upon the degree to which the actual outputs match those that are expected, the testing system will indicate either that the UUT has passed the test (i.e., is functioning properly), or has failed the test (i.e., is not functioning properly).
For purposes of illustration, testing of a computer network routing system using one such conventional stimulus/response testing technique will now be described. In this exemplary arrangement, the system-under-test (SUT) consists of a router which is part of an overall computer network system wherein the router is coupled to a first computer endstation via a first network (e.g., a token ring network) and is also coupled to a second computer endstation via a second network (e.g., a serial type of network connection). Each of the endstations and router may comprise conventional network/computer hardware and software appropriate for permitting communication between the two endstations via the first and second networks and the router. A testing process running on one of the endstations (or, e.g., on a dedicated testing computer coupled to and controlling one or both of the endstations and router via another a third network) initializes the network system for stimulus/response testing (e.g., by initializing test program code/script variables, loading source library functions and variables, binding actual ports/device identifiers supplied via e.g., arguments, into the variables used for same in the program code/script, etc.). Thereafter, the testing process configures the network system (e.g., by getting components' interface addresses, opening the interfaces along the testing path between the testing station and the other endstation, verifying that the interfaces have been properly set up, etc.) so as to permit testing of the SUT and/or specific feature thereof desired to be tested. The testing process then actually begins stimulus/response testing of the SUT by supplying predetermined commands/data to the SUT for causing same to carry out predetermined actions which result in generation of outputs which the process then observes and/or receives as inputs, respectively. For example, in this exemplary prior art arrangement, the testing process may issue commands (e.g., "ping" commands), via the router, for causing the other endstation to transmit data, IP address information, etc. to the testing endstation via the router, and verify that the requested data and/or information has been properly received from the other endstation by comparing these received outputs from the SUT to predetermined, expected outputs therefor, and based upon this comparison, determine whether the SUT is or is not functioning properly. The testing process may also issue commands to the other endstation, via yet another network whose operation is known to be good, for causing the other endstation to undertake stimulus/response testing of the SUT by requesting, via the router, that the endstation that issued the commands transmit data, IP address information, etc. to the other endstation, and determine based upon the received data and/or information whether the SUT is functioning properly. After the testing is completed, the testing process may then close the interfaces previously opened.
Heretofore, in order to carry out a stimulus/response test of a network SUT using the aforesaid type of conventional testing arrangement, an extensive amount of complex test script/program code has had to be written, debugged, and compiled for each such test or series of tests, in order to generate the testing process. As can be readily appreciated, this process places a significant burden upon test programmers in terms of coding time, effort, and frustration. Often, much of the code generated for a specific test (i.e., a test of specific network equipment and/or features) will be substantially identical/generic to code segments that would be necessary to implement many stimulus/response tests (e.g., configuration procedures, input/output comparison test loops, etc.). This leads to much duplication of programming effort, wasted time, and opportunity for introduction of coding errors. Further, use of function libraries and other predefined libraries of commonly used instructions and data have been unable to ameliorate this problem to an extent that would be desirable.
Also, as is the case with much of the program code that is application specific (i.e., written for a specific task and not intended/expected to be reused), large amounts of the testing code being written will lack a significant degree of modularity. This makes editing, debugging and reuse of such code difficult.